Improved Oracle Complexity of Variance Reduced Methods for Nonsmooth Convex Stochastic Composition Optimization

We consider the nonsmooth convex composition optimization problem where the objective isa composition of two finite-sum functions and analyze stochastic compositional variance reducedgradient (SCVRG) methods for them. SCVRG and its variants have recently drawn much atten-tion given their edge over stochastic compositional gradient descent (SCGD); but the theoreticalanalysis exclusively assumes strong convexity of the objective, which excludes several important ex-amples such as Lasso, logistic regression, principle component analysis and deep neural nets. Incontrast, we prove non-asymptotic incremental first-order oracle (IFO) complexity of SCVRG or itsnovel variants for nonsmooth convex composition optimization and show that they are provably fasterthan SCGD and gradient descent. More specifically, our method achieves the total IFO complexityof O((m+ n) log (1/ǫ) + 1/ǫ)which improves that of O(1/ǫ)and O ((m+ n)/√ǫ) obtained bySCGD and accelerated gradient descent (AGD) respectively. Experimental results confirm that ourmethods outperform several existing methods, e.g., SCGD and AGD, on sparse mean-variance opti-mization problem.